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ABSTRACT 

This study v;as designed to determine the extent to 
which the results of an employment survey of graduates of a teacher 
preparation program would have been affected by changes in response 
rate. At the University of Tennessee, Knoxville, a followup of 
teacher education program graduates is conducted annually. A total of 
284 graduates of the 1992 teach*^r education program were identified 
as the target population for tht 1993 survey. A total of 184 
individuals responded to the mail survey (64.8*/0 , with 33.87. 
responding to the first wave, 11 * 27 . responding to the second, 6.3X to 
the third wave, and 7 .^ 7 . responding late. Telephone calls elicited 
information from 40 additional persons (14.1*/0. Supplemental 
information about employment from other sources resulted in the 
eventual determination of the occupations of 265 of the 284 
individuals (93.3*/i). There was no evidence that data collected after 
about 507. of the sample had responded resulted in any meaningful 
differences in survey results. These results suggest that 
concentrating on potential nonresponse bias may not be as important 
as attending to other aspects of survey methodology, such as sample 
size and questionnaire design. (Contains 2 tables and 28 references.) 
(SLD) 



Vc Vc Vc Vc Vc Vc Vc V? V? Vc V? Vc V? Vc V? Vc Vc Vc V? it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 

it it it it it it it it it it it it i; it it it it it it it it it it it it it it it it it it A it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it it 



m 

CO 



CN 

m 

G 

W 



Ox 



ERIC 



u.i. DC^AirrMCNT OF EOOCATiON 
Oft*c« Q* Educational Paaaarch and tmprovamani 

educational resources information 

CENTER (ERIC) 



(^Thi 



Mhia documant naa oean rapioducod as 
(C^aivad Irom tha parson or organization 
originating it 

O Minor changas hava bean mada to improve 
raproduCtion quality 

a Pornts ot viaw or opmions stateo m this docu 
mant do not nacassaniy raprasant oHiCiai 
OERI position or policy 



h-tHMIbblUN lO REHRODUCb THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) ' 



Reexamining Traditional Issues 
in Survey Research: Just How EviS Is 
the Anathema of Low Response Rate? 



by 



Sheldon B. Clark 

Science/Engineering Education Division 
Oak Ridge Institute for Science and Education 
P.O. Box 117 

Oak Ridge, T ennessee 37831 -0117 
and 

Judith A. Boser 

Institute for Assessment and Evaluation 
215 Claxton Addition 
University of Tennessee 
Knoxville, Tennessee 37996-3400 



Paper Presented at the Annual Meeting of 
the American Educational Research Association 
San Francisco, California 
April 18-22, 1995 



2 



BEST COPY AVAILABLE 



Reexamining Traditionai Issues in Survey Research: 

Just How Evii Is the Anathema of Low Response Rate? 

Survey researchers have long been exhorted to strive for high response rates in order to 
maximize the likelihood that the respondents are representative of the population being 
surveyed. It is not surprising, then, that much survey research has been directed 
towards examining the effects of various manipulatable factors (e.g., personalization, 
questionnaire design, commemorative stamps, incentives) on response rate. Because 
such research is so wide-ranging and cuts across so many areas of scientific inquiry, 
many attempts have been made to characterize, condense, quantify, and/or generalize 
the effects of these manipulated variables. Boser and Clark (1993) examined nine of 
these attempts to summarize the results of research on response rates in mail surveys, 
for example. They stopped short of examining the variables that were studied in these 
review articles and the conclusions reached therein, however, partly because the 
populations surveyed and the purposes of the surveys varied so markedly, both within 
and across the studies summarized in the review articles, and partly because of the lack 
of details about the procedures used in some of the integrated reviews. 

It is clear that attempts to reach the goal of minimizing the likelihood of nonresponse 
bias through testing various methods of increasing survey response rates have 
consumed many pages of journal space, generated many papers at professional 
conferences, and fomented more than a few animated discussions among researchers. 
The investigation of the extent of nonresponse bias has taken various approaches: 
comparing respondents with nonrespondents using data already available: comparing 
respondents with a sample of nonrespondents who are interviewed to obtain their 
answers to the items on the questionnaire: comparing responses of early respondents 
with thos? of late respondents, under the assumption that the late respondents are 
similar to nonrespondents and would, in fact, have been nonrespondents had follow-up 
attempts not been instigated: and comparing early respondents with all respondents. 
The variables being compared range from personal characteristics and demographics, 
which are available in records and documents (and sometimes from previous surveys), 
to attitudes and opinions, which must be obtained from the individuals. 

The results obtained in this research have been inconsistent. Some studies have found 
significant differences (although not all were meaningful differences): others have found 
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none. Even when differences have been found in demographics, the variables of 
interest were sometimes not influenced (Finn, Wang & Lamb, 1983). Some researchers 
have suggested that the inconsistent find'ngs in methodological studies in survey 
research may be (and probably are) due to factors that have nothing to do with the 
survey research question ostensibly being addressed. Other researchers are skeptical 
of the generalizability of the findings in many of these studies. For example, Clark 
(1990) states that attempts to identify “magic bullets" in survey research; 

are often self-defeating, and that more attention should be given to the unique 
characteristics of the survey application at hand-- ... the characteristics of the 
target population, availability of resources, . . . [and] the more basic question of 
what the researcher hopes to accomplish by conducting the survey. The primary 
concern of survey research efforts is not always to estimate population 
parameters as precisely as possible. . . Since resources are limited in most 
survey applications, compromises and trade-offs have to be made in such areas 
as sample size; number and type of follow-ups: content, format, and appearance 
of questionnaires: duplication process: and mailing mode. Researchers should 
be more conscientious in considering what they hope to accomplish with a 
particular effort when they make these trade-offs and compromises. They should 
be more creative in developing solutions to their problems by taking advantage of 
whatever special opportunities particular applications may offer, (p.,3) 

Some have suggested that a high response rate is less important when conducting 
surveys of homogeneous populations (Becker, Dottavio & Mengak, 1987: Becker & lliff, 
1983: Wellman, Hawk, Roggenbuck, & Buhyoff, 1980: Leslie, 1972). McDaniel, Madden 
and Verille (1987) recommended that generalizations about nonresponse consider the 
survey topic. Brennan and Hoek (1992) found patterns in the tendency to respond 
across topics in two surveys of the same population. Almost 90 percent of those who 
responded to the first survey also responded to a second survey on a different topic. 
However, more than half of those who did not return forms and just over one-fourth of 
those who refused to participate in the first survey, chose to participate in the 
subsequent survey. 



Despite the plethora of research on variables that affect survey response rates-with the 
implicit goal of determining how to increase response rates-researchers have been 
much less willing to question the premise that high survey response rates are desirable, 
important, and/or even essential, in order to minimize the potential bias introduced by 
the generally unknown (and often unknowable) differences between respondents and 
nonrespondents. This is interesting because, as Dillman points out, "A low response 
rate does not necessarily entail nonresponse error" (1991, p.229). 
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Goudy (1978) and Berdie (1994, 1990, 1989) have proposed that nonresponse bias be 
approached from the standpoint of representativeness of the respondents and that the 
appropriate question is whether the results would have differed substantially had a 
higher response rate been obtained. To answer this question, then, it is necessary to 
compare results obtained from early returns with those derived from the full set of 
returns. Differences between response waves are perceived as being relatively 
unimportant. Mail surveys generally obtain their largest response to the initial mailing, 
so additions to the data from subsequent mailings would seldom have a drastic 
influence on the cumulative results. Goudy found that “minor variations (in variable 
relationships) appear after approximately 50 percent of the sample have completed a 
mail questionnaire and that no differences exist when about 70 percent of those 
contacted have returned data, especially when bivariate relationships are tested.” 

(p. 264) After re-examining 14 studies to compare early returns with total returns, 
Berdie concluded, “As long as response rates exceed 50 to 60 percent, resources used 
to promote response rate beyond that level are usually better spent in other ways.” (p. 
63) 

Hogan (1985), in an examination of high- and low- response rates (67 percent and 36 
percent, respectively) in a survey of graduates from a single institution, concluded that 
there was very little difference in the distributions of the dependent and independent 
variables between the two response rate conditions. She attributed this similarity to the 
homogeneity of the population being surveyed and argued that attempts to increase 
response rates should, nevertheless, be made, simply as a matter of increasing the 
credibility of the research. Berdie (1989) disagreed, concluding that “In most [applied] 
surveys (where the decisions to be made will not be affected by a response variance of 
only a few percentage points), an obsessive fear of nonresponse bias is not justified, 
regardless of whether the population is heterogeneous or homogeneous." (pp. 62-63). 

One homogeneous population that is frequently surveyed consists of college and 
university alumni. (Smith & Bers, 1987). Questionnaires are the major approach to 
obtaining program evaluation information from a specific group of alumni, teacher 
education program graduates (Adams & Craig, 1983). Response rates in mail follow- 
up surveys of teacher education program graduates vary from 10 percent to 100 
percent, with a median of 52 percent (Boser, 1988b). 





In assessing nonresponse bias, college alumni surveys have utilized some of the 
techniques previously described: comparing respondents with a sample of 
nonrespondents (Carifio & Schwedel, 1991; Wilkinson; 1976) and with all 
nonrespondents (Boser 1988a; Hesseldenz, 1976); and comparing early versus late 
respondents or successive response waves (Bowen & Cooper, 1989; Denton, Tsai, & 
Chevrette, 1988; Nielsen, Moos, & Lee, 1978). Yet another approach has been to 
compare the results of two successive alumni surveys with considerably different 
response rates (Hogan, 1985). 




Purpose 

The present study was designed to determine the extent to which the results of an 
employment survey of former graduates of a teacher preparation program would have 
been affected by changes in response rate. It is an extension of the study by Boser and 
Clark (1994) and uses data from the same survey. In the earlier paper, the authors 
concluded that, with respect to the primary purpose of the survey (i.e., to determine how 
many of the graduates of a teacher education program were, in fact, teaching 
approximately a year later), a reliable estimate of the proportion who were teaching was 
reached when the response rate reached about 50 percent (pp. 5-7). This finding is 
consistent with the recommendations of many researchers (e.g., Babbie, 1990), 
including Hogan and Berdie. 

When data relevant to the central purpose of a survey are gathered outside the bounds 
of the questionnaire itself, the researcher must be careful in interpreting other 
information gathered in the survey (i.e., that for which less [or no] information is known 
about the nonrespondents). In interpreting such data, the consequential issue is item 
nonresponse. The degree to which the researcher can have confidence in such data 
depends on the extent of his/her knowledge of the population--through previous contact, 
experience with similar populations, etc. The method used by some researchers (e.g., 
Boser & Clark, 1994; Clark & Nichols, 1983) to construct 100 percent confidence 
intervals around estimares of population characteristics would often yield confidence 
limits that would be too broad to be of much use. In situations in which a researcher 
often deals with the same type of population, however, it is useful to glean as much 
information as possible about them whenever the opportunity presents itself. In this 
spirit, therefore, the authors decided to examine the other (i.e., beyond whether or not 
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they were teaching) characteristics of the respondents to the employment survey, in 
order to further test the 50 percent-response-rate hypothesis. 

Metho d 

At the University of Tennessee, Knoxville (UTK), the follow-up survey of teacher 
education program graduates is conducted annually. One of the major purposes of the 
survey is to determine the employment of graduates, which enables the College of 
Education to provide information to the Career Planning and Placement Services on 
campus about graduates who are still pursuing teaching positions. 

A total of 284 graduates of the 1992 teacher education program were identified as the 
target population for the 1993 follow-up survey. A questionnaire, cover letter, and a 
postage-paid business reply envelope were mailed to each of the individuals on the 
mailing list in mid-October of 1993. Reminder letters (with replacement questionnaires 
and envelopes) were mailed to nonrespondents three weeks later. A final mailing was 
sent to those who had still not responded after another three weeks. Telephone calls 
were made in January 1994 to the remaining nonrespondents. 

In addition to the survey of graduates, other sources of employment information were 
used in attempts to classify those graduates for whom employment data had not been 
obtained. A list of new teachers in the local school system was obtained, and lists of 
nonrespondents were sent to UTK faculty mentoring team leaders. A different survey of 
the same graduates was undertaken in Spring 1994. One follow-up mailing was sent. 
Through the second survey, additional information about nonrespondents to the earlier 
survey was obtained. 

The employment questionnaire contained 42 items, which dealt with the graduates’ 
demographics, their job search, their present employment, their occupational plans, and 
a brief evaluation of the UTK teacher education program. Response options were 
provided for all but seven items. For additional details regarding the survey, the target 
population, the survey procedures, and the initial analytic procedures, the reader is 
refened to Boser and Clark's 1994 paper. 
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Results and Discussion 



A total of 184 individuals responded to the survey by mail, for an overall mail return rate 
of 64.8 percent. There were 96 respondents (33.8 percent) from the first wave, 49 (17.2 
percent) from the second wave, 18 (6.3 percent) from the third wave, and 21 (7.4 
percent) late responses. Telephone calls elicited occupational information for 40 more 
individuals (14.1 percent). An additional 20 graduates (7.0 percent) were identified as 
teachers using the local school system list of new teachers, and UTK faculty members 
provided employment information for 18 individuals (6.3 percent). Three individuals (1.0 
percent) for whom employment information was previously unavailable responded to the 
second survey. When all sources of information had been utilized, occupations of 265 
of the 284 individuals in the target population (93.3 percent) were determined, as well as 
some other information sought on the questionnaire. 

When comparing partial results with final results using inferential statistics, one of the 
assumptions underlying the tests (i.e., independence of the samples) is violated. This 
doesn't mean that the results are not useful, however, particularly in the context of 
survey research, when it is usually difficult, if not impossible, to collect all data on every 
member of the target population. Tables 1 and 2 summarize the results of comparisons 
of responses after the second wave (by which time just over 50 percent of the graduates 
had responded) with those from all available sources. The number of individuals 
responding to each item by the end of the second wave of mailings is indicated by n-,, 
and the number of graduates for whom information for a particular item was available 
from any source (including the first two mailings) is indicated by n 2 . Some items in the 
questionnaire did not apply to all graduates, so the values for n^ and p .2 in Tables 1 and 
2 are not always indicative of item nonresponse. Inasmuch as none of the differences 
between the n-j and n 2 results even approaches statistical significance, there is no 
evidence that data collected after about 50 percent of the target population had 
responded resulted in a meaningful difference in the results. 

As mentioned before, there are deficiencies regarding the statistical approach used in 
this study. The lack of generalizability of the results to heterogeneous populations, to 
populations of a different sort, and to surveys with a different intent and/or content must 
not be trivialized either. On the other hand, in the context in which this particular survey 
was conducted (e.g., purpose, target population, past familiarity with populations of this 
type, necessity of surveying similar populations for similar purposes in the future), this 
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kind of study becomes a useful tool for these researchers-and researchers in 
comparable settings--to make more informed decisions about the allocation of limited 
resources across the various components of the survey process. 

Conclusion 

In applied settings, survey researchers must regularly determine how to allocate limited 
resources across various elements of the survey process. Often--too often, perhaps- 
their primary attention is directed towards battling the universally feared demon of 
potential nonresponse bias, by taking sometimes exotic (and expensive) measures to 
maximize the survey response rate. Unfortunately, these measures come at the 
expense of other factors, such as sample size, questionnaire length, and/or 
questionnaire design. In recent years, some survey researchers have brought into 
question this heretofore sacred tenet of survey research. They have suggested that 
more attention be given to the puroose of the survey, the context in which it is 
conducted, and especially the characteristics of the population being surveyed, rather 
than blindly assuming that their first priority should be to minimize nonresponse. The 
natural extension of such an approach is that survey researchers pay closer attention to 
research-published or otherwise-on populations and purposes that are generally 
consistent with their own. Not only will such an effort more likely result in a less 
confusing array of research findings, but it will also help all researchers who are 
interested in a similar population better understand them, so that more informed 
decisions can be made about the relative importance of maximizing response rates for a 
particular population and a particular purpose. 
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Table 1 



Comparison of Responses After Two Waves with Final Results: Categorical 

Variables 



Focus of Item and Response Options 


"1 


n? 




Prob. 


Classification of graduate 
(3 possibilities) 


145 


265 


1.19, 2df 


0.55 


Major 

(6 i^ssibilities) 


145 


265 


2.29, 5 df 


0.81 


Employed full time? 
(yes, no) 


140 


178 


0.02, 1 df 


0.89 


Employment related to DTK program? 
(yes. somewhat, no) 


137 


174 


0.63, 2 df 


0.73 


Present job situation 

(17 choices, including “other") 


145 


265 


4.41, 15 df 


0.99 


If not teaching but planning to teach, why not now 
(6 reasons, including "other") 


39 


48 


0.55, 6 df 


0.99 


Applied for teacher license? 
(yes. no) 


144 


182 


0.07, 1 df 


0.78 


Did you look for teaching position? 
(yes, no) 


143 


181 


0.06, 1 df 


0.80 


Apply for position in same system you 
interned/student-taught in? 

(yes, no) 


144 


182 


0.14, 1 df 


0.71 


If yes, were you offered position? 
(yes. no) 


97 


119 


0.41, 1 df 


0.52 


Register with DTK placement center? 
(yes. no) 


143 


181 


0.06, 1 df 


0.81 



(table continued on next page) 



NOTES; 

n^ is number of graduates responding to item during fi^st two waves (cumulative response 
rate = 51 .1 percent). 

ng is total number of graduates for whom information was either supplied by the graduate via mail 
survey or obtained from other sources (total response rate = 93.3 percent). 

Some items do not apply to all graduates, so low numbers for n.j and ^2 are not necessarily 
indicativ? of high item nonresponse. 
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Table 1 (continued) 



Comparison of Responses After Two Waves with Final Results: Categorical 

Variables 



Focus of Item and Response Options 


"1 


"2 




Prob. 

^X^ 


Interview on campus? 
(yes, no) 


144 


182 


0.03, 1 df 


0.87 


If yes, were you eventually offered position? 
(yes. no) 


90 


112 


0.41 , 1 df 


0.52 


Notified of any jobs by placement center? 
(yes, no) 


139 


176 


0.22, 1 df 


0.64 


Con' acted by school as a result of being 
registered? 

(yes, no) 


136 


171 


0.00, 1 df 


0.99 


Offered any positions you declined? 
(yes, no) 


143 


181 


0.11, 1 df 


0.74 


If not teaching now, are you seeking teaching job? 
(yes, no) 


54 


64 


0.30, 2 df 


0.86 


If not teaching, do you plan to seek teaching job? 
(yes, no, undecided) 


51 


61 


0.13, 2 df 


0.94 


Plan to be teaching next year? 
(yes, no, undecided) 


144 


182 


0.59, 2 df 


0.75 


Plan to be teaching in 5 years? 
(yes, no, undecided) 


144 


182 


0.21, 2 df 


0.90 


Plan to be teaching in 10 years? 
(yes, no, undecided) 


144 


182 


0.07, 2 df 


0.97 


Plan to teach until retirement? 
(yes, no, undecided) 


144 


182 


0.04, 2 df 


0.98 



(table continued on next page) 



NOTES: 

n-j is number of graduates responding to item during first two vjawes (cumulative response 
rate = 51.1 percent). 

r \2 is total number of graduates for whom information was either supplied by the graduate via mail 
survey or obtained from other sources (total response rate = 93.3 percent). 

Some items do not apply to all graduates, so low numbers for n-j and n£ are not necessarily 
indicative of high item nonresponse. 
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Table 1 (continued) 



Comparison of Responses After Two Waves with Final Results: Categorical 

Variables 



Focus of Item and Response Options 




"2 




Prob. 

^X^ 


Would you major in teacher education again? 
(yes, no, undecided) 


141 


187 


0.26, 2 df 


0.88 


Teaching full-time below postsecondary level? 
(yes, no), a screening item 


143 


181 


0.10, 1 df 


0.75 


College of bachelor's degree 
(6 choices, including "other") 


141 


178 


0.31, 5 df 


0.99 


Transfer status 
(4 choices) 


135 


171 


0.11, 3 df 


0.99 


How did you fulfill field requirement? 
(3 choices) 


144 


182 


0.60, 2 df 


0.74 


Area(s) of licensure 

(22 choices, including “other") 


144 


182 


1.35, 14 df 


0.99 


Gender 
(2 categ ries) 


145 


265 


0.01, 1 df 


0.92 


If not teaching, do you want placement center to 
consider you for teaching positions? 

(yes, no) 


76 


89 


0.05, 1 df 


0.83 



NOTES; 

is number of graduates responding to item during first two v/aves (cumulative response 
rate = 51 .1 percent). 

r >2 is total number of graduates for whom information was either supplied by the graduate via tr.ail 
survey or obtained from other sources (total response rate = 93.3 percent). 

Some items do not apply to all graduates, so low numbers for n.| and D 2 are not necessarily 
indicative of high item nonresponse. 
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Table 2 



Comparison of Responses After Two Waves with Final Results: Continuous 

Variables 



Focus of Item and Response Options 


"1 


"2 


"1 

mean 


"2 

mean 


t 


Prob. 

^Itl 


Quality of academic experiences at UTK 
(1 =poor; 4=excellent) 


143 


181 


2.98 


3.01 


-0.36 


0.72 


Quality of social experiences at UTK 
(1=poor; 4=excellent) 


141 


179 


3.04 


3.07 


-0.52 


0.60 


Quality of cultural experiences at UTK 
(1=poor; 4=excellent) 


142 


179 


2.63 


2.69 


-0.64 


0.53 


Quality of overall experience at UTK 
(1 =poor; 4=excellent) 


144 


182 


3.04 


3.08 


-0.62 


0.53 


Satisfaction with UTK teacher 

preparation program 

(1=very satisfied; 4=very dissatisfied) 


143 


181 


7.00 


7.10 


-0.52 


0.60 


Satisfaction with present employment 
situation 

(1=very satisfied; 4=very dissatisfied) 


142 


180 


1.73 


1.75 


-0.16 


0.88 


Difficulty in obtaining teaching job 
(1=much difficulty; 3=no difficulty) 


92 


120 


2.15 


2.23 


-0.73 


0.47 


Age upon completion of teacher 
education program 
(fill-in-the-blank) 


142 


179 


26.76 


26.78 


-0.03 


0.98 



NOTES: 

is number of graduates res(X)nding to item during first two waves (cumulative response 
rate = 51.1 percent). 

x \2 is total number of graduates for whom information was either supplied by the graduate via mail 
survey or obtained from other sources (total response rate = 93.3 percent). 

Some items do not apply to all graduates, so low numbers for n.j and r \2 are not necessarily 
indicative of high item nonresponse. 
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